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I Abstract. In this paper, to solve the invariant subspace problem, contrac- 

tion operators are classified into three classes ; (Case 1) completely non- 
unitary contractions with a non-trivial algebraic element, (Case 2) completely 
, non-unitary contractions without a non-trivial algebraic element, or (Case 3) 

contractions which are not completely non-unitary. 

We know that every operator of (Case 3) has a non-trivial invariant sub- 
, space. In this paper, we answer to the invariant subspace problem for the 

C**) • operators of (Case 2). Since (Case 1) is simpler than (Case 2), we leave as a 

question. 



Introduction 



o 

An important open problem in operator theory is the invariant subspace prob- 
lem. The invariant subspace problem is the question whether the following state- 
ment is true or not: 

Every bounded linear operator T on a separable Hilbert space H of dimension 
> 2 over C has a non-trivial invariant subspace. 



Since the problem is solved for all finite dimensional complex vector spaces of 



in 

oo 

• dimension at least 2, in this note, H denotes a separable Hilbert space whose 
dimension is infinite. It is enough to think for a contraction T, i.e., ||T|| < 1 on 

• H. Thus, in this note, T denotes a contraction. 
Q\ , If T is a contraction, then 

(Case 1) T is a completely non-unitary contraction with a non-trivial algebraic 
element, or 

(Case 2) T is a transcendental operator; that is, T is a completely non-unitary 
contraction without a non-trivial algebraic element, or 

(Case 3) T is not completely non-unitary. 

In this note, we discuss the invariant subspace problem for operators of (Case 
2). By using fundamental properties (Proposition l2.2l Proposition ^. 31 and Corol- 
lary [231) OI * transcendental operators, we answer to the invariant subspace prob- 
lem for the operators of (Case 2) in Theorem 12.51 

Every transcendental operator defined on a separable Hilbert space H has a 
non-trivial invariant subspace. 

Thus, we answered to the invariant subspace problem for the (Case 2) in 
Theorem 12.51 and, clearly, we know that every operator of (Case 3) has a non- 
trivial invariant subspace. Thus, to answer to the invariant subspace problem, it 
suffices to answer for (Case 1). 
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We do not consider project (Case 1) in this note, and leave as a question; 

Question. Let T G L(H) be a completely non-unitary contraction such that T 
has a non-trivial algebraic element. Then, does the operator T have a non-trivial 
invariant subspace? 

The author would like to appreciate the advice of Professor Ronald G. Douglas. 

1. Preliminaries and Notation 

In this note, C, M and L(H) denote the set of complex numbers, the (norm) 
closure of a set M , and the set of bounded linear operators from H to H where 
H is a separable Hilbert space whose dimension is not finite, respectively. 

For a set A = {m : i G 1} C H , V A denotes the closed subspace of H 
generated by {en : i G I}. 

If T G L(H) and M is an invariant subspace for T, then T\M is used to denote 
the restriction of T to M, and cr(T) denotes the spectrum of T. 

1.1. A Functional Calculus. Let H°° be the Banach space of all (complex- 
valued) bounded analytic functions on the open unit disk D with supremum norm 
[3J. A contraction T in L(H) is said to be completely non-unitary provided its 
restriction to any non-zero reducing subspace is never unitary. 

B. Sz.-Nagy and C. Foias introduced an important functional calculus for com- 
pletely non-unitary contractions. 

Proposition 1.1. Let T G L(H) be a completely non-unitary contraction. Then 
there is a unique algebra representation &t from H°° into L(H) such that : 

(i) < I ) t(1) = Ih, where Ih G L(H) is the identity operator; 

(ii) <& T (g) = T, ifg(z) = z for all z G D; 

(Hi) &t is continuous when H°° and L(H) are given the weak* - 
topology. 

(iv) &t is contractive, i.e. ||$r(u)|| < ||tt|| for all u G H°° . 
We simply denote by u(T) the operator $r(it). 

B. Sz.- Nagy and C. Foias [3] defined the class Co relative to the open unit 
disk D consisting of completely non-unitary contractions T on H such that the 
kernel of <3?t is not trivial. If T G L(H) is an operator of class Co, then 

ker $ T = {u G H°° : u{T) = 0} 

is a weak*-closed ideal of H°°, and hence there is an inner function generating 
ker The minimal function of an operator T of class Cq is the generator 
of ker $t; that is, ker $^ = thtH 00 - Also, mx is uniquely determined up to a 
constant scalar factor of absolute value one pQ . 

1.2. Algebraic Elements. In this section, we provide the notion of algebraic 
elements for a completely non-unitary contraction T in L{H). 

Definition 1.2. [3] Let T G L{H) be a completely non-unitary contraction. An 
element h of H is said to be algebraic with respect to T provided that 0{T)h = 
for some 9 G H°° \ {0}. If h ^ 0, then h is said to be a non-trivial algebraic 
element with respect to T. 

If h is not algebraic with respect to T, then h is said to be transcendental with 
respect to T. 
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2. The Main Results 
If T is a contraction, then 

(Case 1) T is a completely non-unitary contraction with a non-trivial algebraic 
element, or 

(Case 2) T is a completely non-unitary contraction without a non-trivial alge- 
braic element; that is, every non-zero element in H is transcendental with respect 
to T, or 

(Case 3) T is not completely non-unitary. 

It is clear for (Case 3). To answer to the invariant subspace problem for (Case 
2), we provide the following definition; 

Definition 2.1. If T is a completely non- unitary contraction without a non- 
trivial algebraic element; that is, every non-zero element in H is transcendental 
with respect to T, then T is said to be a transcendental operator. 

Proposition 2.2. If T : H — > H is a transcendental operator, then, for any 
9 G H°° \ {0}, 6(T) is one-to-one. 

Proof. Suppose that 6(T) is not one-to-one for a function G H°° \ {0}. Then, 
there is a non-zero element h in H such that 

6{T)h = 0; 

that is, h is a non-trivial algebraic element with respect to T. This, however, is 
a contradiction, since T is a transcendental operator. Thus, 0{T) is one-to-one 
for any 6 6 H°° \ {0}. 

□ 

Recall that an arbitrary subset M of H is said to be linearly independent if 
every nonempty finite subset of M is linearly independent. 

Proposition 2.3. If T : H — > H is a transcendental operator, then, for any 
non-zero element h in H , M = {T n h : n = 0, 1, 2, • • •} is linearly independent. 

Proof. Let h G H \ {0} be given. Suppose that M = {T n h : n = 0, 1, 2, • • •} is 
not linearly independent. Then, there is a polynomial p G H°° \ {0} such that 
p(T)h = 0. Thus h is a non-trivial algebraic element with respect to T. This, 
however, is a contradiction, since T is a transcendental operator. Therefore, M 
is linearly independent. 

□ 

Corollary 2.4. Under the same assumption as Proposition \2.3l for a given func- 
tion 6 G H°° \ {0}, W = {6(T) n h : n = 1, 2, • • •} is linearly independent. 

Proof. In the same way as Proposition 12.31 it is proven. □ 

By a densely defined operator K in H, we mean a linear mapping K on the 
domain T>(K) (which is a subspace of H and dense in H) of K into H [5\. Recall 
that 

(2.1) V{SK) = {x G V{K) : Kx G V(S)}, 

where S and K are unbounded operators [5]. 

Finally, the time has come to answer to the invariant subspace problem for 
operators of (Case 2). 
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Theorem 2.5. If T is a transcendental operator in L{H) and A G o~(T), then 

(i) S = T — XI h has a non-trivial invariant subspace, where In is the identity 
operator on H , and 

(ii) T also has a non-trivial invariant subspace. 

Proof, (i) Since T is transcendental, S ^ (Note that if S = 0, then p(T) = 
where p{z) = z — A), and by Proposition 12.2^ S(= p(T)) is one-to-one. Since S is 
not invertible, S is not onto. Let h be a non-zero element in H such that h does 
not belong to the range of S and 

(2.2) M = \f {S n h : n = 0,1,2,- • •}. 

If M H, then M is a non-trivial invariant subspace for S, and so we assume 
that M = H. If 

(2.3) M' = \j {S n h:n = 1,2,- • •}, 

then we will show that h ^ M' . Since S is one-to-one and h ^ 0, 

(2.4) M' + {0}. 
Suppose that h G M'. 

By Corollary [231 we conclude that {S n h : n = 1, 2, ••■} is linearly independent, 
and so, by Gram-Schmidt process, we have an orthonormal basis B of M' such 
that 

(2.5) B = {Pi(S)h :i = 1,2,- ■■}, 

where Pi(i = 1,2,- • •) is a polynomial satisfying \J{S n h : n = 1, 2, --, m} = 
\J{Pi(S)h : i = 1,2, -,m} for any m€ {1,2,- ■ ■}. 
Then, 

oo 

(2.6) h = ^a t P t (S)h 

i=l 

where dj G C. 
It follows that 

oo 

(2.7) J2 ai Pi(S)h = 

i=0 

where ao = — 1, and Po(^) = Ih- 

For any fc G {0, 1, 2, 3, • • •}, by equation ([277|> . 

oo m m 

(2.8) V aiPAS){S k h) = lim V ai Pi(S){S k h) = lim S fc (V ai Pi{S)h) = 

i=0 i=0 i=0 

By the same way as above, since {S^/i : n = 0, 1, 2, • • •} is linearly independent, 
by Gram-Schmidt process, we have an orthonormal basis B' of H such that 

(2.9) B' = {e i = f i {S)h: i = 0,1,2,- • •}, 

where = 0,1,2,- ■ ■) is a polynomial satisfying \/{S n h : n = 0, 1, 2, ••, m} = 
\J{fi(S)h : i = 0, 1, 2, ••, m} for any m G {0, 1, 2, • • •}. Note that e = pjy. 

Clearly, X^i=o a *^«('^) * s hnear on {c n S n h : c n G C and n = 0,1,2,- • •}, and 
by equation f|2.8j) . ]C^o ^-^U^O ^ s a densely defined operator in if. It is not 
assumed that X^i=o a iPi(S) is bounded or continuous. 
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By equations (12.8H and (12. 9|) . we have that, for any ei G B', 

oo 

(2.10) (^a,P i (,S))e i = 0. 

i=0 

Thus, if 

m 

(2.11) A = {^Cid: a eC, and m = 0,1,2, •• •}, 

i=0 

then, by equation (I2.10p . 

oo 

(2.12) (^a,P i (5))x = 

i=0 

for any x G A C £>(££ OiPi(S)). 
Let 

oo 
i=l 

where is a polynomial such that 

(2.13) Pi(z) = z 9l {z) 

for i G {1,2,3,- • ■}. Note that by the definition of Pj, we can easily find the 
polynomial <?, satisfying equation fl2 . 13f> for i G {1, 2, 3, • • •}. 

Since {S n h : n = 1, 2, 3, • • •} C £>(iT) ( by equation (|2^j> ) and ft G Af ' (defined 
in (12. 3p ) by assumption, i^T is a densely defined operator in H. It is not assumed 
that K is bounded or continuous. 

By equation (I2T121) . since ao = —1, 

oo oo 

(2.14) (SK)(x) = S(J2ai9i(S))(x) = (^a,P(5))(x) = x, 

i=l i=l 

for anyxGic (Note that A C C X>(iT) by equation $TB) [5]). 

Since eo = ii and eo G A, by equation (|2.14j) . we have that SK(eo) = ii. 
Thus, /i belongs to the range of S, but it is a contradiction. 

Therefore, 

(2.15) h i M 1 

Thus, by (|2,4p and (|2.15p . we conclude that M' is a non-trivial invariant sub- 
space for S. 

(ii) In («), T(M') = (5 + AJh)M' C M' 

Therefore, M' is also a non-trivial invariant subspace for T. 

□ 
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